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Abstract 


School districts required under No Child Left Behind to provide supplemental educational 
services (SES) to students in schools that are not making adequate yearly progress rely heavily 
on the private sector to offer choice in service provision. If the market does not work to drive out 
ineffective providers, students will be less likely to gain through their participation in SES. We 
estimate SES provider effects in an urban school district that accounts for a significant share of 
participating students. Using four different estimation techniques, we identify variation in the 
effectiveness of different types of providers, as well as across specific providers, in increasing 
students' math and reading achievement. We expect this research to inform education policy, 
including the reauthorization of Title I and district tutoring interventions both under NCLB and 
following federal waivers from NCLB, and to generally address the viability of policy 
interventions that employ a private market model to improve public sector outcomes. 
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Introduction 


The U.S. No Child Left Behind (NCLB) Act was designed to increase the achievement 
of economically disadvantaged students by introducing greater choice, flexibility and 
accountability in public education. Key legislative provisions include offering educational 
choice to those in persistently low-performing schools; empowering parents with more 
infonnation about the quality of their children's schools; holding states, districts and schools 
accountable for student achievement by identifying and imposing requirements on schools in 
need of improvement, and targeting federal funds on effective practices for improving teacher 
and school quality. 1 As in the decentralization of government functions in other policy areas 
(public assistance, publicly-funded training, child welfare, etc.), the expectation is that 
involving the private sector in public services delivery will bring about a more efficient, 
innovative, competitive, results-oriented and responsive public sector that better meets diverse 
public preferences, values and needs (Frederickson & Smith, 2003; Rivlin, 1992). 

We study the NCLB provision that requires schools that are not making adequate 
yearly progress for three years to offer children in low-income families the opportunity to 
receive extra academic assistance (or tutoring), known as supplemental educational services 
(SES). The primary responsibility for implementing SES lies with school districts, which 
largely rely on the private sector to offer eligible students a range of choices for SES. NCLB 
obligates school districts to set aside 20 percent of their Title I funding for SES and lays out 
criteria and guidelines for state and local educational agencies in approving SES providers 
and arranging for their services. 2 Importantly, it requires state and local educational agencies 

1 See Title I, Section 1 1 16(e) of the Elementary and Secondary Education Act (ESEA), reauthorized by the No 
Child Left Behind Act of 2001. 

2 Title I federal funding, which began in the 1965 Elementary and Secondary Act, was created to allow all 
students an equal opportunity to receive the highest quality education possible. Through Title I, school districts 
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to measure provider effectiveness in increasing student achievement and to use this 
information to withdraw approval of ineffective providers (Heinrich, 2010). 

NCLB also directs school districts to determine eligibility for SES using the same 
infonnation used for making within-district Title I allocations (such as free school lunch 
eligibility), and schools then notify families of their children's eligibility. If more eligible 
students sign up than there are funds available for serving them, districts establish additional 
eligibility criteria, frequently based on student special needs and academic performance. 
However, some eligible students do not follow through in registering for and attending SES, 
and others stop attending before their SES dollar allocation is expended. Therefore, selection 
into treatment (or who gets tutored in SES) and for how long is likely influenced by student 
characteristics, as well as program type and administration (Heinrich, Meyer and Whitten; 
Steinberg, 2011). This makes it enormously challenging for researchers, and even more so for 
school districts with limited resources, to identify effects of SES providers on student 
achievement, while controlling for the effects of other classroom and school interventions. 

Chicago Public Schools (CPS) accounts for a disproportionately large share of the 
students eligible for SES in the U.S., and as we discuss further below, CPS has consistently 
made an effort to evaluate SES effects, including provider effectiveness. We use the most 
recently available longitudinal data from CPS — the 2007-08, 2008-09, 2009-10 and 2010-11 
school years — to estimate provider-specific effects and the effects of different types of SES 
providers using alternative estimation techniques and subsamples and under differing 

can hire teachers to lower student-teacher ratios, provide tutoring for struggling students, create school computer 
labs, fund parent involvement activities, purchase instructional and professional development materials for 
teachers, hire teacher assistants, and more. The 20% Title I set-aside for SES and school transfers cannot be 
spent on administrative costs for these activities, although the district may reallocate any unused set-aside funds 
to other Title I activities after all eligible students have had adequate time to opt to transfer to another school or 
apply for SES. 


3 



assumptions about student and provider selection. Based on prior literature and our conceptual 
framework (discussed below), we hypothesize that there are positive effects of attending SES 
on the achievement of eligible students that will increase with hours attended; provider 
characteristics will be correlated with SES effectiveness, and some specific providers will be 
more effective than others in delivering SES and increasing student achievement. 

We employ four alternative approaches to estimate the effects of SES and different 
type of SES providers on changes in student test scores, while controlling for student selection 
into SES and/or into different provider types. The SES provider types that we examine are: 
district vs. non-district providers, online provision, on-site vs. off-site providers, and for-profit 
vs. not-for-profit (nonprofit or public) provision . 3 We use gains in test scores as our outcome 
in school value-added, student fixed effects, school and student fixed effects, and propensity 
score matching models. We control for school and student time invariant characteristics using 
these four strategies. Each of these modeling approaches makes somewhat different 
assumptions about selection into SES, and as we are estimating different types of effects (e.g., 
provider-specific, provider- type), the analytical samples differ to some extent as well. Thus, 
while we look for overall consistency in the results, we expect some differences as well. 

In general, we find that the CPS district provider delivers significantly more hours of 
tutoring to students who register to receive SES, and there appears to be strong linkage 
between hours of SES attended and increases in student achievement. In addition, the CPS 
district provider is an on-site provider, and we also find that SES delivered by on-site 
providers is effective in increasing student achievement. Students attending SES with online 

3 Online providers use computers/digital technology as the primary format/platform for delivering SES. On-site 
providers offer SES on a public school campus (and typically serve students attending that school), while off-site 
implies that the services are delivered at a site other than the school (e.g., home-based or other location in the 
community). 
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providers and for-profit providers gain less in reading and math as compared to other off-line 
and not-for-profit providers, respectively. We also identify a number of providers that 
consistently have positive and statistically significant effects on student achievement. Overall, 
while average effect sizes are small (approximately 0.09 in reading and 0.06 in math) relative 
to other educational interventions, there is some limited evidence that the SES program 
contributes to improving student achievement. 

In the next section, we discuss our conceptual framework and some background 
infonnation, followed by a review of the literature to date on SES effectiveness. We then 
describe the data and estimation approaches that we employ in the analysis of SES effects and 
present some descriptive statistics on the different types of SES providers and the 
characteristics of CPS students they serve. We follow with a discussion of the results of the 
estimation of SES effects for different provider types and specific providers in CPS and 
conclude with policy implications. 

Conceptual Framework and Background 

Consistent with broader public sector management trends and the classic market 
paradigm, an important component of recent initiatives to improve K-12 education 
effectiveness are efforts to introduce market-driven management and increase pressures on 
educational institutions to develop new strategies for improving student learning and 
educational outcomes. In the context of SES, measuring provider performance and 
disseminating infonnation on provider effectiveness should foster a more competitive market 
for services; contribute to more infonned student choices; encourage innovative approaches to 
service delivery as providers compete for market share, and squeeze out inefficient or 
ineffective providers through choice. 
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Indeed, a large number of diverse organizations with widely varying hourly rates, 
service costs, tutor qualifications, tutoring session length, instructional strategies, and 
curricula have entered the market to compete for the opportunity to provide SES. These 
include national and local organizations, for-profit and nonprofit providers, online and off-line 
providers, those offering services on-site at the schools (and off-site), and in some cases, 
schools districts engaging directly in SES provision. Burch, Steinberg and Donovan (2007: 
121) described the market for SES as "a very new market where hundreds of firms are 
flocking to take advantage of the promise of sizeable revenues.” NCLB explicitly discourages 
states from taking any actions that might limit the supply of providers and range of choices 
available to parents. 4 

Private companies have long been involved in the delivery of K-12 public educational 
services, from textbooks and instructional supports to testing and evaluation, tutoring, and 
more. More recently, the increasing participation of for-profit entities in direct services 
provision through vouchers for school choice, charter schools and mandatory out-of-school 
time interventions has drawn criticism of the prospects for "profiteering" in public education, 
where for-profit Anns are viewed as willing to compromise on quality and to short-change 
students to better their bottom line (Horn, 2011). The primary opposing view counters that 
private firms have considerable potential to cultivate critically-needed innovation in 
educational practice; that is, they are more likely to have the capability and incentives to 

4 The guidance states (U.S. Department of Education, 2005, p. 7): [A state educational agency] that 
desires to set program design parameters should ensure that such parameters do not result in the 
inability of a wide variety of providers, including non- profits, for profits, [local educational agencies] 
and faith-based and community organizations, from being able to participate as eligible providers, 
thereby limiting parental choice. 
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rapidly expand successful practices and approaches, to attract the required financial and 
human capital, and to more cost-effectively deliver educational services. 

In the context of this debate, Peterson (2003) recounts the compromise between 
Capitol Hill conservatives who supported vouchers as a key lever of accountability through 
choice, and liberal politicians who opposed the encroaching private sector role in K-12 public 
education that led to the creation of SES under NCLB. SES allowed for the "back door" entry 
of private providers-for-profit, nonprofit, secular and religious into public schools that were 
failing to make adequate yearly progress, while preserving an important role for school 
districts in arranging access for students to SES and in contracting with private providers. 
School districts identified as “in need of improvement” were, for the most part, prohibited 
from directly providing SES, on the premise that if they were not effective during the regular 
school day, they would be unlikely to do better after school. Private providers, alternatively, 
might benefit from the fact that SES is voluntary rather than compulsory, allowing them to 
potentially work with a more motivated group of students enrolling in an after school 
academic program. In addition, they are free to hire and fire teachers/tutors (unencumbered by 
the typical union rules) and have broad leeway in program structure, focus and curricular 
design. 

School districts have assumed the major responsibility for disseminating information 
on SES providers approved by state educational agencies and operating in the district, 
although to date, this has largely consisted of provider self-reported information on their 
attributes and effectiveness (Heinrich et ah, 2010). SES providers also market their services 
directly to parents and students. Some school district accountability and evaluation units have 
attempted to measure program and provider effectiveness with administrative and student 
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record data, as in Chicago Public Schools. They face important challenges, however, in 
properly evaluating student- and provider-level SES effects, given that participation in SES is 
voluntary among eligible students. 

In general, school districts have few resources for monitoring SES providers and little 
leverage for dismissing them, 5 and they have criticized state educational agencies for their 
lack of responsiveness to reported problems with providers, including fraud and 
ineffectiveness. Chicago Public Schools has been particularly proactive in its efforts to 
disseminate available information to parents and school principals about SES provider 
effectiveness and to develop district policies that support monitoring of providers that use its 
school facilities for service delivery. Still, in the absence of accurate and fairly complete 
infonnation on SES provider performance, states and districts have little capability or 
leverage for disciplining the market (i.e., sanctioning or disqualifying ineffective providers), 
and the benefits of choice in a competitive market are unlikely to be realized if the purchasers 
(eligible students or their parents) have inadequate infonnation for choosing providers. 

Review of Literature on SES Effectiveness 

Previous research on out-of-school-time programs reports mixed findings on the 
effectiveness of these programs in improving student outcomes (Halpem, 2003; Little, 2007). 
Many after-school programs, particularly those with a greater focus on recreational than 
educational activities, have been shown to have minimal effects on students' academic 
progress (Hollister, 2003). SES, however, was designed to explicitly address students' 
educational needs, and the large literature on other after-school/tutoring programs confirms 

5 One of the few provisions available to districts for requesting removal of a provider is following a violation of a 
district policy in use of its buildings/space. 
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their potential to increase student achievement with sufficient hours of tutoring (Dynarski et 
ah, 2004; Halpern, 2003; Lauer et ah, 2006; Little, 2007; Vandell et ah, 2005). Yet to date, 
little is also known about what types or attributes of SES programs are effective, and what 
policies at state or local level can maximize the potential benefits of SES for eligible students. 

Early studies of SES effects on student achievement were primarily descriptive and 
focused on the challenges of implementing the services in an evolving market (Burch et ah, 
2007), while more recent studies have sought to empirically estimate the effects of SES on 
student achievement. Chatterji, Kwon and Sng (2006) estimate the effects of SES in one New 
York school and found small positive effects. Evaluations conducted by Chicago Public 
Schools (CPS) in 2003-04, 2004-05, 2006-2007 and Jones (2009) reported larger gains in 
reading and mathematics for students receiving at least 40 hours of tutoring and for students 
in grades 4-8 who were not English language learners and who received at least 30 hours of 
SES tutoring. A study by the Los Angeles Unified School District (Rickies & Barnhart, 2007) 
found fairly small program effects, attributed primarily to improved perfonnance by 
elementary students. Studies in Minneapolis (Heistad, 2007) and Milwaukee Public Schools 
(Heinrich et ah, 2010), where average SES hours attended are particularly low, did not find 
statistically significant, positive effects of SES participation. Zimmer and co-authors (2007) 
reported average increases in math test score gains of 0.09 standard deviations for students 
attending any SES across the eight sites in their study, and Springer, Pepper and Ghosh (2009) 
likewise estimated increases in test score gains of about 0.09 standard deviations in 
mathematics (and 0.076 standard deviations in reading). In an alternative specification that 
accounted for those who registered but did not attend SES, however, Springer et al. did not 
find statistically significant effects on reading for students attending SES. Springer et al. 
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caution that very few studies rigorously adjust for student selection into SES, identifying only 
four studies besides their own that did (Zimmer, 2006 and 2007; Heistad, 2007; Heinrich et 
ah, 2010). 

There are likewise few studies that rigorously examine the effects of specific SES 
providers on students' academic outcomes (Jones, 2009; Zimmer et ah, 2007; Munoz et ah, 
2008; Ross et ah, 2008; Potter et ah, 2006). This is especially problematic given that it was 
the explicit intent of NCLB to hold providers accountable by giving students and parents the 
necessary information on provider performance to exercise choice and realize the benefits of a 
competitive market for services. 

Jones (2009) used multilevel modeling to explore SES provider effects on students 
who attended SES, controlling for student and school level characteristics. 6 He reported 
moderate effects of attending SES and positive effects of several individual providers in 
Chicago. Studies by RAND (2007) and Socias et al. (2009) used a difference-in-differences 
strategy with value added models to estimate SES provider effects across multiple districts 
after the introduction of district providers in 2006-07. The RAND study found that 
participation in SES had positive effect on students' achievement in reading and math, with 
students participating for multiple years realizing larger gains. Although the RAND study did 
not estimate the effects of specific providers, they did estimate the effects of a district 


6 Our study is distinct from Jones (2009) in that we employ alternative approaches to adjusting for student 
selection into SES at multiple stages (registration, attendance, level of attendance) and also selection into 
particular types of providers. In addition, the CPS studies primarily control for race and gender (although they 
also use gain scores to account for prior learning trajectories) and do not report results on student selection; we 
find other significant predictors of student selection into SES (e.g., student absences from school, prior SES 
attendance, English language learners and students with disabilities, etc.) that provide important information for 
school districts. 
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provider and reported mixed results. Socias et al. (2009) found that the district SES provider 
had no effect on student achievement in Anchorage and Hillsborough. 

The Center for Research on Education Policy (CREP) conducted several, multi-year 
studies in Kentucky, Tennessee and Louisiana on provider-specific effects using a matched 
treatment-comparison strategy. Students attending SES were matched on observable 
characteristics to schoolmates who were eligible for SES but did not participate. They found 
mixed results for the overall effect of attending SES across different states. Specifically, 
Munoz et al. (2008) analyzed student-level achievement for those who attended SES in 
Kentucky and found no significant effects for any individual provider, or for all providers 
combined. Similarly, Ross et al. (2008) found three providers were significantly worse than 
the comparison group in math (suggesting the potential for selection bias that is not 
adequately adjusted through matching), but no effects for any providers in reading for the 
2007-08 school year in Tennessee. Finally, Potter et al. (2006) found most students in 
Louisiana who were served by SES providers did no better or worse than their counterparts 
who were not served. 

The gap in knowledge that we aim to fill is to identify not only effective providers but 
also their attributes and approaches in delivering SES that contribute to their success. This 
should help policy makers better direct the resources that school districts are required to set 
aside for publicly funded tutoring. Although we realize that these results are based primarily 
on a single school district and have limited generalizability, CPS has one of the largest 
numbers of students eligible for and receiving SES, accounting for 10 percent of all SES 
recipients in the nation's public schools in 2008-09 (Center on Innovation and Improvement 
report, U.S. Department of Education). It is also one of a small number of school districts 
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identified as in need of improvement that successfully petitioned the federal government to 
directly provide SES. We more fully explore the implications of direct service provision by 
districts in the discussion of our findings and the concluding section. 


Data and Methods 

We obtained school record data for all students eligible for SES in CPS for the school 
years 2007-08, 2008-09, 2009-10 and 2010-11. The longitudinal database that we constructed 
includes student test scores, demographics, and infonnation on their registration for and 
participation in the SES programs. These data allow us to construct measures of students’ SES 
attendance with specific providers, including the number of hours of SES attended and total 
expenditures from provider invoices. The district also provided infonnation on the SES 
providers, including whether they were online, off-site or on-site; district or non-district; for- 
profit or nonprofit, which allows us to explore the types of organizations and methods of 
service delivery that may contribute to improving student outcomes. 

To construct the key outcome measures of student achievement gains (or changes) in 
student test scores, we use data from standardized tests (Illinois Standardized Achievement 
Tests (IS AT)). For each grade and year, we construct z-scores using the district mean and 
standard deviation, so that the test scores are comparable across grades and years. Table 1 
shows the number of CPS students who are eligible, registered and attended SES for the 
different grades and years as well as the percentage of students with missing scores. Across 
the three panels and school years, the distribution of characteristics for those who are eligible, 
register for and attend SES look similar to the subset of those with gain scores. Therefore, the 
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missing data (other than grade 3) 7 will be treated as random. 

Characteristics of Eligible Students and SES providers 

Table 2 shows the characteristics of students who are eligible and registered for SES 
and who attended SES for the three school years for which we estimate SES effects. These 
measures, along with the student’s grade year and school attended, serve as the core set of 
control variables intended to account for selection into SES in the estimation of SES effects. 
Our choice of these variables reflects specific criteria used by CPS in prioritizing eligible 
students for SES, as well as other characteristics we expect might be associated with the 
likelihood that eligible students will register for and attend SES. Per the law, the primary 
criterion for most school districts is free-lunch eligibility, but as the number of students 
eligible for SES has expanded, districts have specified additional criteria for registrations, 
such as grade level, past perfonnance in school and student special needs. For example, in the 
2009-10 school year, CPS established the following hierarchy of criteria for prioritizing 
registrations among eligible students: (1) free/reduced lunch-eligible; (2) all students in grades 
1-3 and in high school, and English language learners (ELL) and students with disabilities 
(SWD) in grades 4-8, and then (3) students in grades 4-8 by reading stanine. The new 
emphasis on registering SWD and ELL students in 2009-10 is reflected in the larger 
percentage of SWD and ELL students that registered that year (vs. 2008-09), even though the 
percentage eligible were approximately the same from year to year. We also control for 
student absences in the prior school year, as we find, not surprisingly, that students who are 

7 Since the ISAT is taken in grades 3-8 in Illinois, we are not able to include students in grade 3, as there is no 
pre-test information for these students (with the exception of students who were retained in grade 3 in 2008-09). 
In the 2009-10 school year, 6% of students who registered and attended SES are missing (in grades 4 through 8) 
because they don’t have test scores in the prior year (and our outcome measure is gains in achievement). The loss 
of data for 2008-09 school year is lower at 4%. 
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less likely to attend regular school are also significantly less likely to attend SES after school. N 
In addition, students who attended SES in a prior school year are significantly more likely to 
attend SES again. 

If after controlling for the student characteristics shown in Table 2, other factors 
influencing participation in SES are random, then our estimates of the effects of SES should 
approximate the true effects. It is not possible to verify, however, that there are no 
unobserved, selective differences between eligible students who participate in SES and those 
who do not (and serve as comparison group members in our analysis). A well-executed 
random assignment experiment would be needed to assume statistical equivalence between 
students participating in SES and eligible nonparticipants. For example, if there was 
consistently greater demand for SES than funding, school districts might be persuaded to 
allocate opportunities to participate in SES by randomly assigning some fraction of the 
eligible students to receive services and others to a control group of students who would not 
be invited to participate. This would facilitate a causal analysis of the effects of SES on 
student outcomes. In the absence of random assignment, we can only estimate potential 
effects or associations that are suggestive of a possible causal interpretation of the findings, 
which we undertake employing rigorous econometric methods that adjust for selection. 9 


8 Steinberg (2011) finds fewer prior-year disciplinary infractions among SES participants in CPS vs. those who 
did not participate in years 2006-07 and 2007-08. However, as we do not have access to CPS disciplinary data, 
this is an omitted factor in our analysis. 

9 We also explored the possibility of using a regression-discontinuity design for estimating SES effects in CPS, 
given that CPS identified explicit criteria for prioritizing students for SES when the number of eligible students 
exceeded available funding for services. However, after it was discovered in the 2009-10 school year that the 
monies paid out in the prior school year exceeded the funds budgeted for SES, registered students were denied 
services (i.e., told that they could not participate in SES) after being assigned to providers and showing up for 
sessions. As there was no systematic process followed in retracting the offer of services or tracking the students 
who were ultimately refused services, we determined that we could not achieve a clean approach to identification 
of effects through regression-discontinuity analyses. 
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Table 3 shows the characteristics of CPS students by SES provider type in the 2008- 
09, 2009-10 and 2010-11 school years. A few of the notable differences across provider types 
include a higher proportion of ELL students served by the district provider, a lower proportion 
of ELL students served by online providers, and a substantially larger proportion of students 
with disabilities receiving services from off-site providers. Students with disabilities were also 
significantly more likely to attend SES with online providers in 2009-10. These differences in 
student characteristics across different service provider types were also confirmed in the first 
stage propensity score matching model (discussed below) that predicted SES attendance with 
different provider types. 10 
Estimation Strategies 

In our estimation of the average effects of SES and of attending SES with different 
types of providers on changes in student test scores, we adjust for student selection in 
registering for SES and/or their enrollment with particular types of providers (see again Table 
3). In addition, we estimate individual, provider-specific effects for SES providers serving at 
least 30 students in a given school year (and conditional on having data for at least 30 
students). SES providers serving fewer than 30 students are combined in a small-provider 
measure, allowing us to estimate the average effect of smaller providers relative to larger 
ones. We employ the following four strategies to address possible selection bias. 

Value-added model. One way that education studies deal with selection is using 
value-added models. The formal value-added model we employ is specified in equation (1). 
The value-added strategy allows us to control for other classroom and school interventions 

10 Summary statistics on the characteristics of student served by specific (individual) SES providers are available 
upon request from the authors. 
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which are fixed over time, while identifying provider characteristics. For example, if there is a 
reading intervention at a school and those students also attend SES, failing to control for the 
intervention (school fixed effect) would bias the results. The outcome measure is the 
achievement gain made by a given student, which accounts for the possibility that students 
with similar characteristics might enter SES with different underlying achievement 
trajectories (as reflected in their prior test scores). We estimate the following equation, 

Ajst Ajst-i aSESjt "I - pXjt-i T 7t s T Pgt^ - Ejst (1) 

where A Jst is the achievement of student j attending school s in year t; SES/7 is an indicator 
function if the student j attended SES in year /; X Jt -i are student characteristics which include 
student demographics, percent absent in prior year, retained in prior year, and attended SES in 
prior year; jt s is school fixed effect; p gt are grade by year fixed effects, and Ej st is the random 
error tenn. Identification in this specification comes from the average gain in student 
achievement after controlling for student characteristics and school and grade year effects. 

Student fixed effects model. The value-added model assumes that selection depends 
on observed student characteristics. Hence, controlling for them allows us to deal with self- 
selection. However, if selection is on some unobserved or unmeasured characteristics of the 
students, then a value-added strategy could still lead to biased results. The student fixed- 
effects model controls for all time-invariant characteristics of a student, including those that 
are not observed or measured. The following model of an educational production differs from 
equation (1) in that it includes student fixed effects (8,) instead of school fixed effects, 

Aj st = aSESjt + PXjt-i + 5j + p gt + Ej st . (2) 

When we take the first difference of equation (2), we eliminate the student fixed effect (5j), 
and the model estimates the average difference between the gains made by students attending 
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SES with the gains made by similar students in CPS who were likewise eligible for SES. This 
fonnulation imposes some restrictions (or assumptions) that are important to note. First, the 
impact of students’ prior experience does not deteriorate over time. This implies, for example, 
that the effect of the quality of kindergarten has the same impact on student achievement no 
matter the grade. The second assumption is that the unobserved effect of attending SES only 
affects the level but not the rate of growth in student achievement. A concern with this 
restriction is that if students with lower growth are more likely to choose to attend SES, then 
this type of selection may bias the estimates obtained from a gains model. 

In order to relax this restriction, the following equation is estimated, 

Ajst ~ Ajst - 1 = aSESjt + flXjt - 1 + Sj + ngt + Ej s t . (3) 

This approach to estimating the fixed effects model controls for any unobserved differences 
between students that are constant across time. The estimation of this model requires a first 
difference of equation (3) and therefore needs three or more observations for each student. 11 
As students self-select into the SES program, we deal with this selection by using the gain 
scores made by same student in the prior year. Identification of the average impact of SES in 
this model comes from students who participate in one or more but not all years. If these 
students differ in systematic ways from all students who attend SES, then the estimator gives 
a "local" effect (specific to students with these characteristics) instead of an average effect. 

In estimating provider-specific effects, identification comes from students who transfer from 
one SES provider to another over the period of observation. Therefore, it is important that we 
check the robustness of the model results using alternative estimation strategies. Table C.l in 


11 As SES providers serve students at multiple grade levels, it is reasonable to pool information across grade 
levels. 
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Appendix C shows the differences in characteristics between the students who are used for 
identification and those who are not in this estimation approach. 

School and student fixed effects model. The base model for this estimation strategy 
is the combination of the two above methods. A school fixed effect (ns) is added to equation 
(3), which gives: 

Ajst ~ Ajst~ 1 = o.SESjt + [IX/t- 1 + Ks + bj + ngt + Ejst • (4) 

Adding a school fixed effect controls for unmeasured, time-invariant school quality. For 
example, in CPS, school administrators have a role in choosing the providers that deliver 
services on-site at their schools. If principals invite providers that they believe are best suited 
to their students and school environments, provider effects may be correlated with 
unobservable school characteristics that might affect student performance. The inclusion of 
school fixed effects facilitates controlling for time-invariant school characteristics such as 
average school test scores, neighborhood attributes, parental involvement in the school and 
peer composition, to the extent these are unchanging over time. The inclusion of student fixed 
effects effectively controls for student ability and other time-invariant student characteristics. 

As discussed above, identification of the average impact of SES in this model comes 
from students who participate in some but not all years, or in the estimation of provider- 
specific effects, from students who transfer from one SES provider to another, whereas the 
identification of the school effect comes from students who switch schools. This model is 
generally preferred over the value added and the student fixed effects models, as it controls 
for both school and student fixed effects. 12 On the other hand, if the students who switch are 

12 . . 

We also check the results by restricting the analysis to those students who do not change schools and run the 
student fixed effects estimation using Equation (3), and we obtain similar results. 
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different across some time-variant, unobserved characteristics, the results from this strategy 
could still be biased. 

Propensity score matching model. The focus of our analysis using matching methods 
is the estimation of the differential effects of different types of providers for students 
participating in SES. We employ propensity score matching (PSM), a two-step process in 
which the probability of participation in SES (with a particular type of provider) is first 
estimated based on student characteristics (X), generating predicted probabilities of 
participation (propensity scores). The matching process is thereby reduced to a one- 
dimensional problem of comparing students who receive SES from a particular type of 
provider with students with similar propensity scores who participate with other providers. In 
other words, if SES participants and comparison group members have the same propensity 
scores, the distribution of X across these groups will be the same: 

Y 0 J_ D | X => Y 0 ID | P(X), (5) 

and students can be compared on the basis of their propensity scores alone, where D is the 
treatment of attending SES with a given type of provider. 

In applying PSM, we invoke the conditional independence assumption, which implies 
that after controlling for observable characteristics (X), a student’s treatment status is 
unrelated to what his outcome would be in the counterfactual state (Rosenbaum and Rubin, 
1983). The validity of this assumption depends largely on the set of variables or student 
characteristics (X) available for the estimation. There may be some unmeasured factors that 
influence participation with particular types of SES providers; what is important is that 
participation not be predictive of the outcome that would have occurred with another type of 
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provider. In addition, because our outcome variables are defined as the difference between a 
pre-program and post-program measure, we use a panel fonn of the matching estimator 
(difference-in-difference matching) that allows for time-invariant, unobserved differences 
between SES participants and comparison students without biasing estimates of program 
impacts. In estimating this model, we make the assumption that conditional independence 
holds for the periods both before (t) and after (tl) treatment: 

E(Yoti - Y 0t |Di = 1, X) = E(Y 0 ti - Y 0t |Di = 0, X) (6) 

This model estimates the average difference between the gains made by students 
attending SES with a specific type of provider with the gains made by ’’matched” students 
attending with other providers, without putting a functional form on the gain equation (as in 
the case of student fixed effects). The control variables used in the first-stage matching model 
are the same as those included in the other modeling strategies (shown in Table 2). The 
primary PSM matching technique we apply in the second stage model is radius matching, 
which specifies a “caliper ” or maximum propensity score distance (0.01 in our analysis) by 
which a match can be made. It uses not only the nearest neighbor within each caliper, but all 
comparison cases within the caliper (based on the specified distance), and the common 
support condition is imposed to exclude poor matches from the analysis. It is important to 
reiterate that the sample used in this analysis only includes students who attended SES; thus, 
the estimates produced are relative comparisons between providers (or types of providers) that 
show their differential effects on student achievement. 

After-matching balancing tests suggested that the matching generally worked 
effectively; for each of the different estimations (by year and provider type), the covariates 
were fully balanced after matching, with just two exceptions in 2009-10. In the estimation of 
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the effects of on-site providers for students attending SES in 2009-10, English language 
learners were less likely to attend SES with on-site providers (a small difference of 0.013 that 
was not reduced by matching), and in this same year, students with disabilities were 
significantly more likely to attend with off-site providers (a larger difference of 0.165 that was 
likewise not reduced by matching). There were no after-matching balancing concerns 
(statistically significant differences in covariate means) for 2008-09 or 2010-1 1. 

Results of Analyses 

Overall Effects of Attending SES 

Tables 4 and 5 show the average effects of attending SES (using a dummy variable for 
any SES attendance) and the number of hours of SES attended, respectively. Irrespective of 
the variable or estimation strategy used, we find positive and statistically significant results of 
attending SES on math and reading achievement gains for CPS students. The effect size is 
very similar using the student fixed effects or school and student fixed effects strategies in 
reading and math. Table 4 shows that the effect size for reading is approximately 0.07-0.09 
standard deviations, and for math, it is about 0.05-0.06. These effect sizes are about one-third 
of the average annual reading and math gains for elementary and middle school students (as 
reviewed by Hill et al., 2008). 13 

Table 5 reports the effects of the number of hours of SES received on student 
achievement in math and reading. We find that there is a positive and statistically significant 
effect of an additional hour of SES on student achievement. Appendix A shows histograms of 

13 Hill et al., 2008 find an average annual reading gain for 5 th -6 th graders of about 0.32 standard deviations and of 
0.23-0.26 standard deviations for 6 th -8 th graders. 
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the number of hours of SES received by students who attended SES in these three school 
years (2008-09, 2009-10 and 2010-11). These histograms show distinct spikes in the 
distribution of SES hours attended, typically close to 40 and 60 hours of SES attended, 
reflecting, in part, that the number of hours attended is a function of the rate the providers 
charge and the maximum dollars allocated per student by CPS. For the average number of 
hours of SES received (approximately 40 hours), estimated effect sizes are comparable to 
those shown in Table 4. In Appendix B, we report findings on SES effects at the 40- and 60- 
hour attendance thresholds (using PSM methods) only for those students who attended SES 
(and for brevity, only for school year 2008-09). In general, these results are consistent with 
prior studies (Lauer et al., 2006), which suggest effect sizes are larger for programs offering 
45 or more hours of tutoring. 

In addition, we also estimated generalized propensity score models 14 using data on the 
total number of hours of SES students attended over the years 2008-2011 to assess the 
cumulative effects of tutoring on students’ math and reading achievement (for those tutored). 
We are not able to attain balance across student characteristics at all intervals (e.g., quartiles 
or deciles) of hours of SES attended (likely due to unmeasured selection into different levels 
of SES attendance), and thus, we view these results as illustrative rather than causal. The 
graphical display of these results in Appendix B (Figures B.l and B.2) suggests a linear 
relationship between hours tutored and reading gains through about 55 hours of tutoring, with 
diminishing returns to additional hours of SES setting in around 60 hours of tutoring. 


14 See Hirano and Imbens (2004) for more details on their extension of propensity score matching methods to 
cases where the treatment is continuous. In generalized propensity score matching, a “dose-response function” is 
estimated, where in this example, the “dose” is the number of hours a student is tutored, and the “response” is the 
impact that a given level of tutoring has on their reading and math gains. 
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Alternatively, the results for students’ math achievement suggest a steady, positive 
relationship between hours tutored and math gains through 80-plus hours. 

Heterogeneous Effects of SES Providers 

A diverse range of providers come and go from the tutoring market. If there are 
identifiable attributes of the more effective providers and this information is made available to 
key stakeholders in SES (e.g., state and local educational agencies, students and parents), it 
could have the potential of increasing the effectiveness of SES (and tutoring services more 
generally) over time. We first discuss results that compare the CPS district provider with other 
(non-district) SES providers. Under federal regulations, school districts that have been 
identified for improvement are not eligible to provide SES, but CPS was one of a small 
number of districts granted waivers by the U.S. Department of Education to offer SES. 
Coming policy changes may allow an increasing number of school districts to engage in direct 
provision of SES, as well as other flexibility for innovative approaches to service provision. 

There are several possible reasons we might expect to observe a different effect size 
for the district provider (A.I.M. High) compared to other SES providers. First, CPS uses only 
regular school-day teachers as tutors in its program, with the intent to provide continuity in 
learning from the regular school day to after-school instruction, as well as to take advantage 
of teacher knowledge about student needs. That said, if the instruction is less likely to be 
innovative or just more of the same from the school day, this feature might not benefit 
students. In addition, we know from our analysis of infonnation on SES provider rates in CPS 
and other large urban districts that CPS not only charges a relatively low hourly rate as a 
district provider (approximately $28 per hour in 2009-10), but it also appears to have 
influenced rate-setting among other (non-district) providers in the Chicago area. We observe 


23 



the same non-district providers operating in other districts charging as much as twice the rate 
they charged for an hour of their services in CPS. Because the hourly rate charged directly 
affects the number of hours of SES students can receive before reaching the district maximum 
per-student allocation, CPS students are attending more hours of SES (compared to students 
in other districts). In this context, it is possible that a district provider could contribute to 
higher hours of SES tutoring received, and correspondingly, to greater program effectiveness. 

The first analysis includes both SES-eligible students who attended and those who did 
not attend SES in the sample, so that the estimated effects are for district providers and other 
(non-district) providers relative to outcomes for eligible students who did not receive SES. 
(Alternatively, when we restrict our sample to include only students who attended SES, our 
estimated effects are differential effects between the district and other providers). The value- 
added (with school fixed effects), the student fixed effects and school with student fixed 
effects results (see Table 6) all show statistically significant effects of CPS district-provided 
services on students’ math and reading achievement relative to students who do not receive 
SES. The coefficients are the changes (measured in standard deviations from district average 
reading and math test scores) in an average student’s outcome that can be expected if the 
student participates in SES. The estimated coefficients are consistently larger for district vs. 
nondistrict providers, suggesting that attending SES with a district provider may generate a 
larger effect on student achievement than attending with a non-district provider. We 
explicitly test this in additional analyses discussed below and presented in Table 11. Although 
there are a few differences in estimated effect sizes (estimates from the fixed effects 
estimation approaches are slightly larger), they generally represent about one-third the 
average annual student gain scores. 
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Tables 7 and 8 present the provider-specific effect sizes for the district provider and 
other providers who have at least 30 students attending SES with them; smaller providers 
(serving less than 30 students) are grouped into a single “small provider” indicator. There is 
fairly strong agreement among the estimates of value-added and both types of fixed-effect 
models results, with a handful of providers standing out as particularly effective in both sets 
of results and/or across the three years (i.e., the district provider-A.I.M. High, Newton 
Learning, Orion’s Mind, School Service Systems and SES of Illinois). With few exceptions, 
the provider-specific effect sizes from the school and student effects models are very close to 
those of the student fixed effects models. 

In Appendix B, we report an average effect size of approximately 0.06 that was 
statistically significant for SES providers delivering 40 or more hours of tutoring to students. 
The CPS district provider was getting significantly more hours of SES to the students it 
served (an average of 48 hours, and nearly twice as many as other providers). Thus, it is not 
surprising that the effect size of the district provider is approximately twice the size of the 
average for all providers. 15 Effect sizes for Newton Learning, School Service Systems and 
SES of Illinois are similarly large. These providers have different hourly rates, although 
because we view the hourly rate and number of hours as part of the treatment (as defined or 
designed by a given provider), we do not include number of hours attended as a covariate. 

Tables 9 and 10 report average effects of the district and other provider types on CPS 
students’ reading and math achievement in 2008-09, 2009-10 and 2010-11, estimated using 
value-added models, student fixed effects models, and school and student fixed effects 
models. Table 1 1 shows the differential effects of different types of providers for those 

15 To test this hypothesis, we interacted the district indicator with the number of hours of SES the students 
received, and the difference between district and non-district provider disappeared. 
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students who attended SES using the three above mentioned estimation techniques and 
propensity score matching (respectively). Even though the samples differ to some extent 
(particularly for Table 11, where students attending with different types of providers are 
compared), the results are fairly consistent across specifications. On-site providers 
consistently have positive effects on student math and reading achievement (effect sizes of 
0.042-0.081), and students attending SES with the district provider generally outperfonned 
other on-site providers (although the difference between district and nondistrict providers is 
not statistically significant, as verified by an F-test, Prob > F = 0.1 52). 16 The results also 
suggest that online providers are generally less effective than other providers. The difference 
in the coefficients of online vs. on-site are statistically significant at 5% significance level. 

The sample size for off-site providers was relatively small in 2008-09 and 2009-10 (75 and 91, 
respectively), although more than 2,000 students attended SES with off-site providers in 
2010-11. In 2009- 1 0, off-site providers served a larger percentage of S WD students, and thus, 
we suggest interpreting these results with some caution. 1 7 

The effect sizes of attending SES with for-profit providers versus other nonprofit or 
district providers (see Table 10) suggest that for-profit providers are generally less effective 
than district/public providers in increasing student achievement, particularly for math. 

Students attending with for-profit providers gain about 0.03 standard deviations less than the 
district providers in reading and about 0.07 standard deviations less in math. Table 3 shows 

16 In each of the estimations reported, the standard errors of the coefficients are largest for the school and student 
fixed effects models. 

17 CPS prioritized students with disabilities in 2009-2010, with the result that even though the proportion of 
SWD among eligible students is the same in 2008-09 and 2009-10, the proportion attending SES is nearly double 
in 2009-10. We speculate that this compositional change in a small subsample might explain the differences in 
the magnitude of off-site effects (as seen in Table 9) across the school years. 
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that only about 2.5% of students attended SES with a nonprofit provider, and another 10-16% 
attended with the district provider; clearly, the largest share of students attend SES with for- 
profit providers. 

The results in Table 1 1 show the estimated differential effects between the district and 
other providers, as well as the effects of other provider types, from the analysis that only 
includes students who attended SES. As the analysis is done for only those students who 
attend SES (controlling for their selection into specific provider types in the PSM analyses), a 
smaller sample size leads to larger standard errors than in the fixed effects strategies and 
fewer statistically significant results. The differential impacts can also be calculated from the 
previous tables, and the results are consistent across these different estimation strategies. 

The first four rows of results in Table 1 1 present the average differential effect 
between the district provider and other SES providers serving CPS students in 2008-09, 2009- 
10 and 2010-11, estimated using value-added models, student fixed effects models, school 
and student fixed effects models, and propensity score matching (respectively). The results 
suggest that, on average, students attending SES with the district provider gain more on 
reading tests (approximately 0.02-0.04 standard deviations more where statistically 
significant) and more on math tests (approximately 0.02-0.06 standard deviations) than 
students attending with non-district providers. We also restricted the sample to only students 
who attended SES with an on-site provider, either the district or another on-site SES provider, 
to detennine if the district provider performance differed from that of other on-site providers. 
These findings (in the fifth to eighth rows of results in Table 1 1) suggest that the district 
provider generally outperfonned other on-site providers, with students who attended SES on- 
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site with the district provider realizing larger gains of 0.04-0.06 standard deviations more 
(where statistically significant) than other on-site providers. 

Results of analyses comparing providers that deliver SES instruction online with other 
(off-line) providers (see Table 1 1) suggest that online providers are generally less effective 
than other providers, although the coefficients (differential effect sizes) are statistically 
significant only for the 2008-09 school year and in two specifications. The estimated 
(negative) differential effects for 2008-09 are highly comparable between the value-added and 
PSM models, suggesting students attending with online providers gain approximately 0.03- 
0.04 less than students attending SES with other providers. A final set of results comparing 
effect sizes of attending SES with for-profit providers versus other providers (nonprofit or 
public) again suggests that for-profit providers are generally less effective than non-profit and 
district/public providers in increasing student achievement, particularly for math. 

Conclusion 

Supplemental educational services (out-of-school tutoring) are a core provision of 
NCLB, in which school districts are mandated to pay for the cost of provision of after-school 
tutoring for low income and disadvantaged students who attend schools that are not making 
adequate yearly progress for three years. A key feature of this mandate is its reliance on the 
private sector to offer eligible students greater choice in a competitive market that is expected 
to encourage innovative service approaches and squeeze out ineffective providers. Identifying 
tutoring provider effects on student achievement is essential to generating the infonnation 
necessary for students and parents to make informed choices of tutoring providers, but efforts 
to estimate provider effects are complicated by the fact that participation is voluntary. In this 
paper, we have drawn on nonexperimental methods to estimate the effects of SES providers 
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on student achievement in a large urban school district (Chicago Public Schools), which 
accounts for a significant share of students receiving tutoring under NCLB. 

The findings of our empirical analyses of the effects of SES providers who served 
eligible CPS students in the 2008-09, 2009-10 and 2010-11 school years suggest that there is a 
statistically significant effect of attending SES on student achievement, particularly for those 
who receive at least 40 hours of tutoring. These effect sizes represent about one-third of the 
annual gains made by students in these schools, and the gains from tutoring generally increase 
with the number of hours of tutoring received. 

Additionally, we find that the district provider is more effective than non-district and 
other on-site SES providers in increasing the math and reading test scores of students who 
attend SES, although we recognize that this effect may not generalize beyond CPS. In an 
ongoing qualitative component of our study that involves multiple, large urban school districts 
(Heinrich et ah, 2012), we have identified several distinctive features of the implementation of 
SES in CPS that might explain the district’s greater effectiveness, including the significantly 
lower hourly rate charged that allows more hours of tutoring for students, the use of regular 
school day teachers as tutors, and the use of school-based SES coordinators in monitoring and 
coordinating the delivery of tutoring services. Alternatively, we find that students receiving 
tutoring from online providers appear to gain less in reading and math than students attending 
with other providers. We are currently further documenting and analyzing the different types 
of digital/online providers (vs. non-digital providers), in an effort to identify empirically and 
qualitatively what contributes to these providers’ lower average effectiveness, (e.g., do the 
online sessions involve interactions with a live tutor, or are they pre-loaded, self-directed 
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sessions?) 18 This is particularly important given that some urban school districts have seen 
substantial expansions of online providers’ market shares of students attending SES. 19 

Students receiving tutoring from for-profit providers gain less than those attending 
with non-profit providers or the district provider, particularly in math. We also identified 
individual SES providers that were significantly more effective in producing math and reading 
gains for CPS students. These are a mix of for-profit and nonprofit SES providers, although 
each of them offers SES on-site (at the public schools that students attend). 

Given that unmeasured differences in students who attend SES or attend with 
particular types of providers could still introduce bias in these results, we are encouraged by 
the fact that the findings are fairly consistent across the four different rigorous estimation 
methods that make different assumptions. We also note that our findings on provider-specific 
effects are consistent with the most recent CPS evaluation of SES as well (Jones, 2009), 
which identified many of the same providers as being among the most effective in CPS. 
Finally, we believe that our research has identified some basic characteristics of more 
effective approaches to the organization and management of SES programs that might be 
considered by other school districts seeking to improve tutoring outcomes, although we again 
caution that our findings should be viewed as associations (rather than causal effects). 

By design, if NCLB or its successor initiatives (under recently granted waivers to 
states from NCLB provisions) are to achieve the broader goal of reducing the academic 

18 This has turned out to be a much larger undertaking than originally anticipated, because of the extent of 
variation among digital providers in practice and discrepancies in materials used by providers to advertise 
services or seek state approval for SES provision. 

19 

In our ongoing, multisite study, we have observed online tutoring companies with a student “market share” as 
high as 88 percent in one urban school district, and a single digital provider delivering tutoring to more than 
14,000 students in another large, urban school district. 
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achievement gap through after-school tutoring for students in under-perfonning schools, 
tutoring program administrators need adequate and independent (not self-reported) 
infonnation on provider effectiveness to guide students' and parents' choices. We have 
presented readily adaptable estimation strategies that can be used by states and school districts 
to generate infonnation on tutoring provider effectiveness for students and parents that will 
help them to make better informed choices. Whether under NCLB or waivers from its 
provisions, many school districts across the country will continue to spend millions of Title I 
funds on tutoring for economically and academically disadvantaged students, and these 
findings will help to inform those who are looking for guidance in improving these services 
and their impacts on student achievement. 

Finally, the different levels of SES program administration — primarily district and 
state — could improve their coordination in oversight and monitoring of tutoring provider 
perfonnance to more fully realize the potential of the competitive market in improving student 
outcomes. For example, with the evidence from this and related studies, states and districts 
might introduce performance-based contracts to exert more control over provider rates, 
minimum levels of tutoring delivered and other parameters of service delivery that this 
research suggests could contribute to improved outcomes. Indeed, under newly granted state 
waivers from NCLB provisions, some school districts are already establishing maximum 
hourly rates and/or other requirements that will ensure students are offered a minimum of 40 
hours of tutoring (Heinrich et ah, 2012). In this regard, we expect our study to more broadly 
speak to the viability of education and other policy interventions that employ a private market 
model to improve public sector outcomes. 
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Table 1: Number of students eligible, registered for SES and who attended SES, with and 
without gain scores, for the 2008-09, 2009-10 and 2010-11 school years 


Year 2008-09 

District mean adjusted gain scores 





All students 

Students with gain scores 

Students w/missing gain scores % 


Eligible 

Registered 

AttendedSES 

Eligible 

Registered 

AttendedSES 

Eligible 

Registered 

AttendedSES 

Grade 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 







3 

13,363 

7,252 

6,530 

1497 

870 

757 

N/A 

N/A 

N/A 

4 

11,823 

5,849 

5,279 

11,336 

5,605 

5,070 

4% 

4% 

4% 

5 

11,581 

5,308 

4,755 

11,149 

5,126 

4,602 

4% 

3% 

3% 

6 

13,088 

5,567 

4,921 

12,629 

5,379 

4,753 

4% 

3% 

3% 

7 

12,695 

4,441 

3,818 

12,263 

4,292 

3,697 

3% 

3% 

3% 

8 

12,698 

4,350 

3,742 

12,297 

4,219 

3,635 

3% 

3% 

3% 

Total 

75,248 

32,767 

29,045 

61,171 

25,491 

22,514 

4% 

4% 

3 % 

Year 2009 

10 

District mean adjusted gain scores 




All students 

Students with gain scores 

Students w/missing gain scores % 


Eligible 

Registered 

AttendedSES 

Eligible 

Registered 

AttendedSES 

Eligible 

Registered 

AttendedSES 

Grade 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 







3 

16,739 

8,030 

7,652 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

4 

14,380 

3,052 

2,839 

13,142 

2,876 

2,684 

9% 

6% 

5% 

5 

13,912 

2,726 

2,515 

12,738 

2,571 

2,381 

8% 

6% 

5% 

6 

14,182 

2,687 

2,448 

13,078 

2,536 

2,314 

8% 

6% 

5% 

7 

14,074 

1,804 

1,618 

12,991 

1,715 

1,543 

8% 

5% 

5% 

8 

14,255 

1,737 

1,532 

13,178 

1,626 

1,435 

8% 

6% 

6% 

Total 

87,542 

20,036 

18,604 

65,127 

11,324 

10,357 

8% 

6% 

5% 

Year 2010-11 

District mean adjusted gain scores 




All students 

Students with gain scores 

Students with missing gain scores % 


Eligible 

Registered 

AttendedSES 

Eligible 

Registered 

AttendedSES 

Eligible 

Registered 

AttendedSES 

Grade 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 

Freq. 







3 

18,607 

7,537 

4,748 

1493 

737 

596 

N/A 

N/A 

N/A 

4 

17,181 

3,073 

2,423 

16,475 

2,961 

2,384 

4% 

4% 

2% 

5 

17,256 

2,523 

1,994 

16,541 

2,453 

1,965 

4% 

3% 

1% 

6 

16,905 

2,337 

1,790 

16,242 

2,259 

1,754 

4% 

3% 

2% 

7 

15,780 

1,437 

1,062 

14,992 

1,352 

1,015 

5% 

6% 

4% 

8 

16,201 

1,442 

1,033 

15,346 

1,371 

996 

5% 

5% 

4% 

Total 

101,930 

18,349 

13,050 

81,089 

11,133 

8,710 

4% 

4% 

2% 
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Table 2: Student characteristics of those who are eligible, registered and attended SES 



All Eligible 
Students 

Eligible students 
w/gain scores 

All Registered 
Students 

Registered 
students w/gain 
scores 

All Attended 
Students 

Attended students 
w/gain scores 

Student characteristics 

Year 08-09 

75,248 

61,171 

34,838 

25,492 

30,306 

22,515 

Asian 

i% 

i% 

i% 

0% 

i% 

0% 

Black 

53% 

54% 

59% 

61% 

58% 

59% 

Hispanic 

44% 

44% 

39% 

38% 

40% 

39% 

White 

2% 

2% 

1% 

1% 

1% 

1% 

Other race 

0% 

0% 

0% 

0% 

0% 

0% 

% Female 

49% 

49% 

50% 

50% 

50% 

50% 

% ELL 

12% 

10% 

14% 

11% 

15% 

11% 

% FRL 

100% 

100% 

100% 

100% 

100% 

100% 

% Sp. Ed 

14% 

14% 

15% 

15% 

15% 

15% 

Attended SES last year 

26% 

28% 

38% 

40% 

39% 

41% 

% Absent last year 

6% 

5% 

5% 

5% 

5% 

5% 

Retained this year 

4% 

2% 

4% 

1% 

4% 

1% 

Read Gain (dist) zscore 

0.00 

0.00 

0.03 

0.03 

0.03 

0.03 

Math Gain (dist) zscore 

0.00 

0.00 

0.03 

0.03 

0.04 

0.03 


All Eligible 
Students 

Eligible students 
w/gain scores 

All Registered 
Students 

Registered 
students w/gain 
scores 

All Attended 
Students 

Attended students 
w/gain scores 

Student characteristics 

Year 09-10 

87,542 

65,414 

20,036 

11,393 

18,604 

10,424 

Asian 

2% 

i% 

i% 

i% 

i% 

i% 

Black 

49% 

48% 

49% 

49% 

48% 

49% 

Hispanic 

47% 

48% 

46% 

47% 

47% 

47% 

White 

2% 

2% 

2% 

2% 

2% 

2% 

Other race 

0% 

0% 

2% 

1% 

2% 

1% 

% Female 

49% 

49% 

47% 

45% 

47% 

45% 

% ELL 

12% 

10% 

20% 

18% 

20% 

18% 

% FRL 

100% 

100% 

98% 

98% 

98% 

98% 

% Sp. Ed 

13% 

14% 

22% 

29% 

22% 

30% 

Attended SES last year 

42% 

42% 

58% 

58% 

59% 

60% 

% Absent last year 

4% 

4% 

4% 

4% 

4% 

4% 

Retained thisyear 

2% 

1% 

4% 

1% 

4% 

1% 

Read Gain (dist) zscore 

-0.01 

-0.01 

0.05 

0.05 

0.06 

0.06 

Math Gain (dist) zscore 

-0.02 

-0.02 

0.02 

0.02 

0.03 

0.03 


All Eligible 
Students 

Eligible students 
w/gain scores 

All Registered 
Students 

Registered 
students w/gain 
scores 

All Attended 
Students 

Attended students 
w/gain scores 

Student characteristics 

Year 10-11 

101,930 

81,089 

18,349 

11,133 

13,050 

8,710 

Asian 

2% 

2% 

i% 

i% 

i% 

i% 

Black 

42% 

43% 

45% 

47% 

43% 

44% 

Hispanic 

53% 

52% 

51% 

49% 

53% 

52% 

White 

2% 

2% 

2% 

2% 

2% 

2% 

Other race 

1% 

1% 

1% 

1% 

1% 

1% 

% Female 

49% 

49% 

48% 

48% 

48% 

47% 

% ELL 

16% 

13% 

26% 

21% 

26% 

21% 

% FRL 

100% 

100% 

100% 

100% 

100% 

100% 

% Sp. Ed 

12% 

13% 

16% 

21% 

18% 

21% 

Attended SES last year 

8% 

10% 

9% 

15% 

11% 

16% 

% Absent last year 

5% 

5% 

5% 

5% 

4% 

4% 

Retained thisyear 

2% 

3% 

5% 

8% 

5% 

8% 

Read Gain (dist) zscore 

0.01 

0.01 

0.09 

0.09 

0.10 

0.10 

Math Gain (dist) zscore 

0.01 

0.01 

0.09 

0.09 

0.10 

0.10 
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Table 3: Characteristics of Chicago Public Schools students served by different types of SES 
providers in the 2008-09, 2009-10 and 2010-11 school years 


Provider Type 

District 

Non-district 

Online 

On-site 

Off- site 

For-profit 

Nonprofit 

Number of students, Year 2008-09 

4037 

21455 

4166 

19377 

91 

20770 

626 

Asian 

0% 

0% 

1% 

0% 

0% 

0% 

0% 

Black 

53% 

62% 

70% 

61% 

66% 

62% 

63% 

Hispanic 

46% 

36% 

28% 

37% 

34% 

36% 

36% 

White 

1% 

1% 

1% 

1% 

0% 

1% 

1% 

Other race 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

% Female 

50% 

50% 

51% 

50% 

49% 

50% 

54% 

% English language learners 

13% 

11% 

7% 

11% 

7% 

11% 

7% 

% Free lunch eligible 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

% Students w/disabilities 

16% 

15% 

15% 

15% 

27% 

15% 

16% 

Attended SES last year 

41% 

39% 

40% 

40% 

42% 

39% 

38% 

% Absent last year 

5% 

5% 

5% 

5% 

5% 

5% 

5% 

Retained this year 

1% 

1% 

1% 

2% 

0% 

1% 

2% 

Read Gain (dist) zscore 

0.040 

0.020 

0.000 

0.030 

0.050 

0.020 

0.020 

Math Gain (dist) zscore 

0.060 

0.020 

0.000 

0.030 

0.000 

0.020 

0.050 

Number of students, Year 2009-10 

1182 

10142 

1105 

9757 

75 

9886 

255 

Asian 

0% 

1% 

2% 

1% 

25% 

1% 

0% 

Black 

48% 

50% 

53% 

50% 

41% 

50% 

57% 

Hispanic 

49% 

46% 

41% 

46% 

31% 

46% 

38% 

White 

1% 

2% 

2% 

2% 

3% 

2% 

1% 

Other race 

1% 

1% 

2% 

1% 

0% 

1% 

4% 

% Female 

44% 

45% 

43% 

45% 

37% 

45% 

39% 

% English language learners 

20% 

17% 

14% 

17% 

17% 

17% 

14% 

% Free lunch eligible 

98% 

98% 

98% 

98% 

100% 

98% 

96% 

% Students w/disabilities 

34% 

29% 

38% 

28% 

47% 

29% 

37% 

Attended SES last year 

42% 

44% 

46% 

44% 

33% 

44% 

43% 

% Absent last year 

4% 

4% 

4% 

4% 

4% 

4% 

5% 

Retained this year 

1% 

1% 

0% 

1% 

0% 

1% 

2% 

Read Gain (dist) zscore 

0.100 

0.050 

0.040 

0.050 

0.320 

0.050 

0.110 

Math Gain (dist) zscore 

0.070 

0.020 

0.000 

0.020 

-0.060 

0.020 

0.060 

Number of students, Year 2010-11 

1133 

7331 

1069 

7114 

1417 

7262 

69 

Asian 

1% 

1% 

1% 

1% 

1% 

1% 

0% 

Black 

39% 

45% 

80% 

46% 

38% 

45% 

52% 

Hispanic 

57% 

51% 

19% 

51% 

57% 

51% 

48% 

White 

2% 

2% 

0% 

2% 

3% 

2% 

0% 

Other race 

2% 

1% 

0% 

1% 

1% 

1% 

0% 

% Female 

46% 

47% 

48% 

48% 

44% 

47% 

54% 

% English language learners 

26% 

20% 

9% 

21% 

20% 

20% 

20% 

% Free lunch eligible 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

% Students w/disabilities 

23% 

23% 

18% 

22% 

24% 

23% 

13% 

Attended SES last year 

14% 

17% 

18% 

17% 

15% 

17% 

20% 

% Absent last year 

4% 

4% 

5% 

4% 

4% 

4% 

4% 

Retained this year 

9% 

8% 

11% 

8% 

8% 

8% 

12% 

Read Gain (dist) zscore 

0.148 

0.091 

0.073 

0.095 

0.115 

0.091 

0.085 

Math Gain (dist) zscore 

0.140 

0.086 

0.074 

0.100 

0.072 

0.087 

0.012 
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Table 4: The average effect of attending SES (under alternative estimation strategies) 



Reading 

Value Added Model (with School Fixed Effects) 

Student Fixed Effects 
Model 

School and Student 
Fixed Effects Model 

Year 2008-09 

Year 2009-2010 

Year 2010-11 

Coef SE 

Coef SE 

Coef SE 

Coef SE 

Coef SE 

Attended SES 

0.043 0.006 

0.094 0.009 

0.075 0.009 

0.078 0.013 

0.078 0.013 

No. of observations 

61171 

63506 

80510 

205187 

205187 

No. of schools 

227 

454 

302 

466 

466 

No. of students 

61171 

63506 

80510 

119970 

119970 


Math 

Value Added Model (with School Fixed Effects) 

Student Fixed Effects 
Model 

School and Student 
Fixed Effects Model 

Year 2008-09 

Year 2008-09 

Year 2010-11 


Coef SE 

Year 2009-2010 

Coef SE 

Coef SE 

Coef SE 

Attended SES 

0.046 0.005 

0.053 0.008 

0.064 0.009 

0.057 0.012 

0.057 0.012 

No. of observations 

61464 

63773 

80614 

204094 

204094 

No. of schools 

227 

455 

302 

466 

466 

No. of students 

61464 

63773 

80614 

119441 

119441 
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Table 5: Average effects of the number of hours of SES attended on student achievement in math and reading 


Hours of SES 
Attended 

Reading 

Value Added Model (with School Fixed Effects) 

Student Fixed Effects 
Model 

School and Student 
Fixed Effects Model 


Year 2008-09 

Year 2009-2010 

Year 2010-11 

Coef SE 

Coef SE 

Coef SE 

Coef SE 

Coef SE 

Attended SES 

0.0011 0.00013 

0.0026 0.0002 

0.0016 0.0002 

0.0020 0.0003 

0.0028 0.0005 

No. of observations 

61171 

63506 

80510 

205187 

205187 

No. of schools 

227 

454 

302 

466 

466 

No. of students 

61171 

63506 

80510 

119970 

119970 


Math 

Hours of SES 
Attended 

Value Added Model (with School Fixed Effects) 

Student Fixed Effects 
Model 

School and Student 
Fixed Effects Model 

Year 2008-09 

Year 2009-2010 

Year 2010-11 


Coef SE 

Coef SE 

Coef SE 

Coef SE 

Coef SE 

Attended SES 

0.0013 0.0001 

0.0013 0.0002 

0.0015 0.0002 

0.0016 0.0003 

0.0012 0.0005 

No. of observations 

61464 

63773 

80614 

204094 

204094 

No. of schools 

227 

455 

302 

466 

466 

No. of students 

61464 

63773 

80614 

119441 

119441 
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Table 6: The average effects of attending SES with district provider vs. a non-district provider 



Reading 


Value Added Model (with School Fixed Effects) 

Student Fixed Effects 
Model 

School and Student 
Fixed Effects Model 

Year 2008-09 

Year 2009-2010 

Year 2010-2011 

Coef SE 

Coef SE 

Coef SE 

Coef SE 

Coef SE 

District 

0.045 0.013 

0.128 0.019 

0.091 0.020 

0.111 0.027 

0.106 0.026 

Nondistrict 

0.042 0.006 

0.088 0.009 

0.072 0.010 

0.073 0.014 

0.074 0.014 

No. of observations 

61171 

63506 

80510 

205187 

205187 

No. of schools 

227 

454 

302 

466 

466 

No. of students 

61171 

63506 

80510 

119970 

119970 


Math 

Value Added Model (with School Fixed Effects) 

Student Fixed Effects 
Model 

School and Student 
Fixed Effects Model 

Year 2008-09 

Year 2009-2010 

Year 2010-2011 

Coef SE 

Coef SE 

Coef SE 

Coef SE 

Coef SE 

District 

0.067 0.011 

0.065 0.020 

0.092 0.022 

0.123 0.032 

0.115 0.031 

Nondistrict 

0.037 0.005 

0.047 0.008 

0.059 0.009 

0.048 0.013 

0.048 0.013 

No. of observations 

61464 

63773 

80614 

204094 

204094 

No. of schools 

227 

455 

302 

466 

466 

No. of students 

61464 

63773 

80614 

119441 

119441 
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Table 7: Effects of attending SES with the district provider and other providers on reading gains 



Value Added Model (with School Fixed Effects) 

Student Fixed Effects 

School and Student 


Year 2Q08-09 

Year 2009-2010 

Year 2010-2011 

Model 

Fixed Effects Model 

Reading Gain 

Coef 

SE 

Co ef 

SE 

Co ef 

SE 

Co ef 

SE 

Co ef 

SE 

District 

0.045 

0.013 

0.129 

0.019 

0.091 

0.020 

0.110 

0.027 

0.105 

0.026 

Small Providers 

0.033 

0.038 

0.174 

0.048 

0.082 

0.075 

0.120 

0.060 

0.119 

0.060 

A+ Tutoring Service, LTD 

0.095 

0.044 





0.153 

0.098 

0.147 

0.101 

Academic Advantage 




0.107 

0.045 

-0.047 

0.134 

-0.045 

0.137 

ASPIRA 

0.009 

0.006 





-0.029 

0.069 

-0.024 

0.067 

Babbage Net School 

0.043 

0.034 

0.081 

0.082 

-0.020 

0.073 

0.025 

0.055 

0.037 

0.056 

Black Star Project 

0.089 

0.069 

0.051 

0.122 



0.018 

0.136 

0.035 

0.135 

Brain Hurricane 

0.080 

0.024 

0.040 

0.022 

0.107 

0.025 

0.101 

0.055 

0.090 

0.053 

Brainfuse One-to-One 

-0.005 

0.031 

0.044 

0.060 

0.256 

0.075 

0.007 

0.069 

0.012 

0.070 

Brilliance Academy 

0.061 

0.047 



0.009 

0.030 

0.054 

0.072 

0.067 

0.070 

Cambridge Educational 

0.050 

0.028 

0.094 

0.041 

0.016 

0.032 

0.032 

0.056 

0.023 

0.057 

Chess Academy 

0.034 

0.029 

-0.030 

0.026 

0.114 

0.040 

0.063 

0.040 

0.059 

0.041 

Children's Ho me+Aid Soc. 

0.007 

0.020 

0.044 

0.099 

0.038 

0.113 

0.025 

0.097 

0.035 

0.099 

ClubZ! Tutoring Service 

0.048 

0.047 

0.126 

0.083 

0.026 

0.046 

0.060 

0.077 

0.057 

0.078 

CSC Julex Learning 

0.051 

0.051 









Educate Online 

0.022 

0.016 

0.194 

0.049 



0.055 

0.048 

0.055 

0.048 

Failure Free Reading 

-0.029 

0.062 









Huntington - ONSITE 

0.049 

0.020 

0.035 

0.047 

0.127 

0.038 

0.073 

0.055 

0.074 

0.055 

1 HP (ONSITE) 

0.102 

0.037 

0.013 

0.059 

0.031 

0.085 

0.097 

0.112 

0.096 

0.113 

Imagine Learning 




-0.066 

0.080 





Learn it Systems 




0.232 

0.085 





Literacy for All 

-0.004 

0.033 

0.082 

0.038 

0.052 

0.076 

0.115 

0.058 

0.112 

0.058 

Mainstream Development 

0.057 

0.059 

0.033 

0.069 



0.026 

0.111 

0.037 

0.112 

NESI 

0.000 

0.063 



0.017 

0.071 

0.046 

0.053 

0.027 

0.040 

Newton Learning 

0.053 

0.013 

0.115 

0.019 

0.069 

0.022 

0.126 

0.028 

0.124 

0.027 

One to One 

0.164 

0.068 









Orion's Mind 

0.044 

0.011 

0.069 

0.015 

0.094 

0.023 

0.072 

0.024 

0.074 

0.025 

Platform Learning 

-0.012 

0.024 





-0.023 

0.058 

-0.013 

0.061 

PoderSer (ONSITE) 

0.093 

0.035 

0.113 

0.088 



0.059 

0.136 

0.080 

0.134 

Princeton Review 

0.038 

0.017 





0.024 

0.054 

0.034 

0.054 

Progressive Learning 

0.045 

0.018 

0.082 

0.030 

-0.022 

0.034 

0.027 

0.034 

0.029 

0.034 

Rocket Learning Partners 

0.025 

0.029 

0.058 

0.033 

-0.025 

0.038 

0.029 

0.049 

0.035 

0.050 

School Service Systems 

0.059 

0.018 

0.130 

0.032 

0.074 

0.033 

0.111 

0.046 

0.109 

0.047 

SES of Illinois 

0.044 

0.025 

0.128 

0.019 

0.056 

0.023 

0.095 

0.034 

0.098 

0.036 

SPC Educational Services 




-0.053 

0.048 





Smart Kids, Inc 




0.216 

0.188 





Spanish Learning Center 




0.135 

0.064 





Sylvan Learning 




0.168 

0.081 





The Homework Master Center 



0.222 

0.086 





Tutorial Services 

0.036 

0.042 





-0.038 

0.181 

-0.042 

0.184 

Unparalleled Solutions 

0.065 

0.045 

0.102 

0.037 

0.061 

0.048 

0.050 

0.081 

0.058 

0.084 

No. of observations 

61171 

63506 

80510 

205187 

205187 

No. of schools 

227 

454 

302 

466 

466 

No. of students 

61171 

63506 

80510 

119970 

119970 


42 


Table 8: Effects of attending SES with the district provider and other providers on math gains 


Math Gain 

Value Added Model (with School Fixed Effects) 

Year 2008-09 

Year 2009-2010 

Year 2010-2011 

Coef 

SE 

Coef 

SE 

Coef 

SE 

District 

0.065 

0.011 

0.066 

0.020 

0.092 

0.022 

Small Providers 

0.032 

0.036 

0.000 

0.044 

0.080 

0.099 

A+ Tutoring Service, LTD 

0.021 

0.058 





Academic Advantage 





0.166 

0.040 

ASPIRA 

0.022 

0.097 





Babbage Net School 

0.063 

0.028 

-0.032 

0.025 

-0.005 

0.083 

Black Star Project 

0.049 

0.054 

0.128 

0.111 



Brain Hurricane 

0.071 

0.025 

0.061 

0.033 

0.039 

0.027 

Brainfuse One-to-One 

-0.022 

0.022 

-0.032 

0.071 

0.000 

0.051 

Brilliance Academy 

0.000 

0.058 



0.002 

0.044 

Cambridge Educational 

0.090 

0.031 

0.051 

0.035 

0.063 

0.031 

Chess Academy 

0.066 

0.039 

0.065 

0.025 

0.102 

0.046 

Children's Home+Aid Soc. 

0.173 

0.021 

0.086 

0.037 

0.024 

0.146 

ClubZ! Tutoring Service 

-0.017 

0.039 

0.047 

0.071 

0.083 

0.015 

CSC Julex Learning 

-0.029 

0.060 





Educate Online 

0.022 

0.013 

0.041 

0.044 



Failure Free Reading 

0.027 

0.044 





Huntington - ONSITE 

0.016 

0.016 

0.037 

0.036 

0.035 

0.029 

IEP (ONSITE) 

0.037 

0.049 

0.020 

0.038 

0.132 

0.093 

Imagine Learning 





0.051 

0.067 

Learn it Systems 





0.029 

0.089 

Literacy for All 

-0.003 

0.035 

0.092 

0.031 

0.037 

0.043 

Mainstream Development 

-0.012 

0.040 

0.033 

0.044 



NESI 

0.071 

0.038 



0.118 

0.024 

Newton Learning 

0.061 

0.016 

0.031 

0.017 

0.071 

0.023 

One to One 

0.157 

0.083 





Orion's Mind 

0.050 

0.010 

0.037 

0.013 

0.081 

0.017 

Platform Learning 

-0.018 

0.031 





PoderSer (ONSITE) 

0.007 

0.072 

-0.010 

0.034 



Princeton Review 

0.050 

0.017 





Progressive Learning 

0.008 

0.017 

0.046 

0.023 

-0.005 

0.037 

Rocket Learning Partners 

0.005 

0.024 

-0.013 

0.034 

-0.021 

0.044 

School Service Systems 

0.012 

0.023 

0.106 

0.022 

0.051 

0.037 

SES of Illinois 

0.046 

0.026 

0.086 

0.023 

0.053 

0.021 

SPC Educational Services 





-0.015 

0.091 

Smart Kids, Inc 





-0.132 

0.013 

Spanish Learning Center 





0.079 

0.031 

Sylvan Learning 





0.158 

0.107 

The Homework Master Center 




0.012 

0.105 

Tutorial Services 

-0.005 

0.047 





Unparalleled Solutions 

0.072 

0.032 

0.026 

0.041 

0.096 

0.046 

No. of observations 

61464 

63773 

80614 

No. of schools 

227 

455 

302 

No. of students 

61464 

63773 

80614 
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Table 9: Effects of attending SES with the district, online, on-site and off-site providers 


Reading 

Value Added Model (with School Fixed Effects) 

Student Fixed 
Effects Model 

School and 
Student Fixed 

Provider type 

Year 2008-09 

Year 2009-2010 

Year 2010-2011 

Coef 

SE 

Coef 

SE 

Coef 

SE 

Coef 

SE 

Coef 

SE 

District 

0.044 

0.013 

0.128 

0.019 

0.091 

0.020 

0.102 

0.024 

0.096 

0.024 

Online 

0.007 

0.011 

0.033 

0.024 

- 0.009 

0.025 

0.001 

0.023 

0.001 

0.023 

Onsite 

0.042 

0.006 

0.081 

0.009 

0.066 

0.011 

0.069 

0.015 

0.070 

0.015 

Offsite 

0.082 

0.073 

0.350 

0.077 

0.089 

0.020 

0.134 

0.050 

0.135 

0.051 

No. of obs. 

61171 

63506 

80510 

205187 

205187 

No. of schools 

227 

454 

302 

466 

466 

No. of students 

61171 

63506 

80510 

119970 

119970 

Math 


Value Added Model (with School Fixed Effects) 

Student Fixed 
Effects Model 

School and 
Student Fixed 

Provider type 

Year 2008-09 

Year 2009-2010 

Year 2010-2011 

Coef 

SE 

Coef 

SE 

Coef 

SE 

Coef 

SE 

Coef 

SE 

District 

0.064 

0.011 

0.065 

0.020 

0.092 

0.022 

0.106 

0.028 

0.098 

0.028 

Online 

-0.010 

0.010 

-0.001 

0.018 

- 0.027 

0.025 

- 0.023 

0.023 

- 0.023 

0.023 

Onsite 

0.038 

0.005 

0.049 

0.008 

0.064 

0.010 

0.050 

0.013 

0.050 

0.013 

Offsite 

-0.001 

0.059 

- 0.064 

0.053 

0.056 

0.020 

0.025 

0.057 

0.031 

0.058 

No. of obs. 

61464 

63773 

80614 

204094 

204094 

No. of schools 

227 

455 

302 

466 

466 

No. of students 

61464 

63773 

80614 

119441 

119441 
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Table 10: Effects of attending SES with the district, for-profit and not-for-profit providers 


Reading 

Value Added Model (with School Fixed Effects) 

Student Fixed 
Effects Model 

School and 
Student Fixed 

Provider type 

Year 2008-09 

Year 2009-2010 

Year 2010-2011 

Coef 

SE 

Coef 

SE 

Coef 

SE 

Coef 

SE 

Coef 

SE 

District 

0.045 

0.013 

0.129 

0.019 

0.091 

0.020 

0.151 

0.034 

0.142 

0.034 

For-profit 

0.042 

0.006 

0.089 

0.009 

0.071 

0.010 

0.120 

0.028 

0.117 

0.029 

Not-for-profit 

0.054 

0.025 

0.099 

0.061 

0.030 

0.059 

0.056 

0.030 

0.053 

0.030 

No. of obs. 

61171 

63506 

80510 

205187 

205187 

No. of schools 

227 

454 

302 

466 

466 

No. of students 

61171 

63506 

80510 

119970 

119970 



Math 

Value Added Model (with School Fixed Effects) 

Student Fixed 
Effects Model 

School and 
Student Fixed 

Provider type 

Year 2008-09 

Year 2009-2010 

Year 2010-2011 

Coef 

SE 

Coef 

SE 

Coef 

SE 

Coef 

SE 

Coef 

SE 

District 

0.067 

0.011 

0.065 

0.020 

0.092 

0.022 

0.162 

0.035 

0.151 

0.035 

For-profit 

0.037 

0.005 

0.048 

0.008 

0.058 

0.009 

0.099 

0.025 

0.096 

0.026 

Not-for-profit 

0.063 

0.031 

0.036 

0.036 

-0.071 

0.068 

0.066 

0.027 

0.063 

0.027 

No. of obs. 

61464 

63773 

80614 

204094 

204094 

No. of schools 

227 

455 

302 

466 

466 

No. of students 

61464 

63773 

80614 

119441 

119441 
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Table 11: Differences in SES effects by provider types (for students who attended SES) 



Reading 

Math 


Year 2008-09 

Year 2009-10 

Year 2010-11 

Year 2008-09 

Year 2009-10 

Year 2010-11 


Coef 

SE 

Coef 

SE 

Coef 

SE 

Coef 

SE 

Coef 

SE 

Coef 

SE 

District vs. Non-district 

Value Added 

0.020 

0.018 

0.027 

0.030 

0.040 

0.035 

0.039 

0.017 

-0.003 

0.027 

0.024 

0.027 

Student FE 

0.033 

0.089 

Results are for all 3 years 

0.063 

0.099 

Results are for all 3 years 

School&Student FE 

0.008 

0.094 

0.062 

0.102 

PSM 

0.029 

0.012 

0.065 

0.020 

0.042 

0.020 

0.041 

0.011 

0.049 

0.018 

0.048 

0.018 

District vs. other onsite providers 

Value Added 

0.016 

0.019 

0.032 

0.032 

0.060 

0.037 

0.040 

0.018 

-0.014 

0.026 

0.026 

0.033 

Student FE 

0.025 

0.082 

Results are for all 3 years 

0.070 

0.101 

Results are for all 3 years 

School&Student FE 

0.005 

0.089 

0.065 

0.106 

PSM 

0.016 

0.011 

0.059 

0.019 

0.041 

0.019 

0.033 

0.010 

0.043 

0.017 

0.046 

0.021 

Online vs. Not online 

Value Added 

-0.028 

0.013 

0.025 

0.029 

-0.014 

0.034 

-0.042 

0.012 

-0.005 

0.020 

-0.021 

0.033 

Student FE 

-0.087 

0.089 

Results are for all 3 years 

-0.047 

0.081 

Results are for all 3 years 

School&Student FE 

-0.087 

0.095 

-0.049 

0.086 

PSM 

-0.027 

0.011 

-0.009 

0.022 

0.012 

0.022 

-0.038 

0.010 

-0.024 

0.019 

-0.040 

0.021 

For-profit vs Non-profit 

Value Added 

-0.022 

0.015 

-0.028 

0.032 

-0.036 

0.032 

-0.041 

0.016 

0.016 

0.022 

-0.007 

0.025 

Student FE 

-0.022 

0.089 

Results are for all 3 years 

-0.053 

0.090 

Results are for all 3 years 

School&Student FE 

0.006 

0.093 

-0.048 

0.093 

PSM 

-0.009 

0.011 

-0.067 

0.020 

-0.046 

0.020 

-0.024 

0.011 

-0.055 

0.017 

-0.038 

0.018 
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Appendix A 


Figure 1 : Hours of SES received by students attending SES in 2008-09 school year 



Figure 2: Hours of SES received by students attending SES in 2009-10 school year 



Figure 3: Hours of SES received by students attending SES in 2010-11 school year 
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Appendix B: Analyses of SES Effects for Students Who Attended Some SES 

Table B.l below reports the estimated effects of SES (measured as changes in students math and 
reading scores) at common peaks of SES attended (40 and 60 hours) for elementary and middle 
school students. We compare only students who attended in the 2008-09 school year, separately 
matching students with attendance levels above and below each of these points and controlling for 
their probability of registering for and attending SES and other characteristics. 

For elementary school students (first column of results), we find comparably sized effects for both 
math and reading of approximately 0.06 standard deviations (range: 0.054 - 0.68). In comparison, 
these effect sizes are about one-fifth the size of the average annual reading and math gains in 
elementary students (Hill et ah, 2008). For middle school students, the estimated effect sizes are 
statistically significant only in the case of math (0.053-0.067). As average annual math gains trend 
downwards as grade level increases (Hill et ah, 2008), these effects are substantively greater than 
for elementary students (about one-fourth of the average annual gains in math). 


Table B.l: The average effect of attending SES from propensity score matching analysis 


Year 2008-09 

Reading 

Math 


Grades 

3-5 

Grades 

6-8 

Grades 

3-5 

Grades 

6-8 


Coefficient 

Coefficient 

Coefficient 

Coefficient 

Attend 40 or more hours of 
SES vs <40 hours 

0.068** 

0.018 

0.054** 

0.067** 

Attend 60 or more hours of 
SES vs <60 hours 

0.058** 

0.022 

0.057** 

0.053** 


** Statistically significant at 0.5% 


Figures B.l and B.2: Generalized propensity score estimates of cumulative SES effects on 
student reading and math achievement 
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Appendix C 


Table C.l: Number of students who either changed status or changed SES providers 


Changed 

Changed providers 


SES 

status 

0 

1 

Total 

0 

54,425 

4,852 

59,277 

1 

0 

24,494 

24,494 

Total 

54,425 

29,346 

83,771 


Note: “Changed SES status” is a dummy variable that takes the value 1 if the student attended SES last year but not this 
year (and vice-versa). “Changed providers” is a dummy variable that indicates if the student changed providers. 


Table C.2: Summary of students who either changed status or changed SES providers 



Changed SES 
status 

Changed Providers 

No 

Yes 

No 

Yes 

# of obs 

59,277 

24,494 

54425 

29,346 

black 

46% 

53% 

46% 

53% 

Hispanic 

50% 

45% 

51% 

45% 

Female 

49% 

49% 

50% 

48% 

English language learner status 

7% 

11% 

6% 

12% 

Free lunch status 

100% 

100% 

100% 

100% 

Students w/disabilities status 

12% 

18% 

10% 

20% 

% Absent last year 

5% 

4% 

5% 

4% 

Retained last year 

1% 

2% 

1% 

2% 


Note: “Changed SES status” is a dummy variable that takes a value of 1 if the student attended SES last year but not 
this year (and vice-versa). “Changed providers” is a dummy variable that indicates if the student changed providers. 


51 




